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ABSTRACT 

A study assessed the effects of metacognitive 
training in question answering strategies on students' ability to 
answer postreading comprehension questions. In the first part of the 
study,, baseline data concerning the level of metacognitive awareness 
exhibited by expert readers when answering questipns were gathered, 
through the use of 44 skilled adult reeders. In the second part of 
the study, 100 fourth, sixth, and eighth grade students of varying 
reading abilities were randomly assigned to . experimental or control 
groups. Those students in the experimental condition were trained to 




g r _oup 

readers in identifying questions by_type.,_selecting-an appropriate 
question answering, strategy, and providing a complete and accurate 
-respons e""t"6 questions. Performance was better on text-based rather 
than knowledge-based questions, with ability and developmental levels 
of the students affecting their performance in predicted directions. 
(FL)„ 
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The Effect of Metacognitive Training on 

Children's Question-Answering Behavior I 

This study represents an attempt to integrate two distinct lines of 
research: the role of questions as facilitators of comprehension and the 
role of metacognition, specif ical ly as a means of increasing one's knowledge 
and flexible selection of learning strategies. 

Questions, pervasive in the school environment, occur in textbooks, 
diagnosis and assessment procedures, and in classroom discussions. The 
question, often cited as a teaching technique (e.g., Socratic dialogue), is 
more often the means for determining what information a reader has gained 
from text. Labels such as skilled and unskilled reader often are based upon 
students' performances on these post reading. comprehension questions. 

Research in the area of questions reflects concerns in three related 
areas. First, several researchers (e.g., Barrett, 1976; Pearson & Johnson, 
1978) have developed taxonomies of questions that presumably reflect sotoe 
hierarchy of levels of cognitive activi ty. Second, researchers (e.g., 
-Guszak, 1966; Bartolome, 1969; Chou-Hare & Pulljam, I98O) have examined the 
frequency of occurrence of questions from the various categories. Finally, 
the facilitative effect of questions on learning from or memory for textual 
information has been studied (e.g., Rothkopf, 1966; Frase, 1 968; Anderson & 
Biddle, 1975}- Considered together, the literature suggests (a) that a 
number of question categories exist, each requiring different cognitive 
activity or strategy use to locate correct response information; (b) that 
questions of a literal, or at least text-based nature, dominate the 
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school environment, and (c) that questions°do facilitate one's learning from 
and memory for text. 

Beyond their frequency and utility, questions can also reveal informa- 
tion about how the reader processes and learns from text (Kavale S Schreirier 
1979-80; Raphael, Winograd,^ s Pearson, I98O; Andre' S "Anderson, 19J8-79). 
By integrating the more traditional approaches to questions with a growing 
body of research concerning readers' awareness of and control over processes 
engaged in while reading (known as metacogni t ion) , researchers have been 
able to study the relationship between students 1 performances on various 
types of questions and their awareness of and control over those strategies 
used in. answering questions. Thus, one question to be considered concerns 
our ability to teach children to use the available strategies or techniques 
for locating appropriate response informal ion. That is, can we teach 
children to better cope with the questions they must so often face? 

The second line of research, metacogni tion, provides the framework 
Within flhich the answers t'o_this- question. can^ba^soughL.-^lThe..roJe_ of. me.ta-^ 
cognition in research into questions stems from an increasing emphasis 
upon the active role of the learner or the reader. The term metacogni tive 
knowledge has two distinct aspects: (a) knowledge of cognition--the 
"awareness of one's own mental piocesses and abilities, and (b) regulation 
of cognition — those mechanisms that allow one to evaluate one's progress 
during the learning activity (Brown, 1931). Thus, in terms of question 
asking or question answering skills, the influence of metacogni tive research 
has been to move question research beyond the issues of what kind of 
questions are asked and how questions have been asked to the issues of 
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what strategies students use to answer questions and when and where those 
strategies apply. » 

Within the framework of metacogni tion, training studies investigating 
the use of strategies in a number of areas have been conducted. The purpose 
of this work has been to evaluate whether the use of strategies facilitates 
the appropriate cognitive activity and whether students can be trained to 
employ these strategies spontaneously. Most of this research has investi- 
gated developmental differences in strategy use on memory tasks (e.g., 
cBrown, Campione, S Murphy, 1977; Chi, I98O; Flavell, 1970). ^Differences 
that occur have been attributed to either production or ta mediation \ 
deficiencies (Flavell, 1970) or to differences in prior knowledge (Chi, 
1980). Production deficiencies occur when^ the learner, ei the r knows, or 
has the potential for learning when to use, a particular strategy but fails 
to invoke the strategy spontaneously, Mediation deficiencies occur when 
the learner does not invoke an appropriate strategy because of capacity 
limitations rather than a potentially correctable lack of strategy knowl- 
edge, Jhe results of these training studies indicate that students repre- 
senting a range in ages (K-Adult) and aptitude (retarded to normal) can 
be taught to use a variety of memory mnemonics successfully. 

More relevant to the study of question answering strategies are the 
results from studies of strategy use during reading. Several researchers 
have suggested that the use of strategies may be one factor which differ- 
entiates the skilled frt>m the less skilled reader (Golinkoff, 1975-76; 
Guthrie S.Tyler, 1976; Ryan, I98I; Weinstein S Rabinovitch, 1971). As a 
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result of the findings concerning strategy use, researchers havt uegun to 
train readers of different ages and ability levels to increase their aware- 
ness and use of metacogni t i ve strategies involved in question-generation 
(Andre' s Anderson, 1978-79; Morse, 1976; Smith, 1973) as well as question 

Answering (Raphael ; et a 1~~, 1980). ^ ~" 

^ Developing awareness and training strategy use involves a'number of 

factors. Brown, Campione, and Day ( 1 98 1 ) have developed instructional 
guidelines which take several of these factors into account. The guidelines 
include training an instruct ional 1 y relevant skill, providing feedback on 
performance, and giving explicit instruction regarding the applicability of 
the skill. The procedure is based upon the assumption that students should 
■* be informed participants in any strategy trailing ski 11 "program to maximize 
both the efficient use of a strategy and the transfer of its use to other 
situations. The present study uses this model in an attempt to provide 
.an effective method for teaching students of different developmental 
and ability levels relevant strategies for responding differentially to 
situations in which questions, task demands, and available resources vary. 

Specifically, we set out to evaluate the effect of providing students 
with a* plan for directing their cognitive activity based upon their deci- 
sions about the interrelationships among a'question, the available text 
information, and one's background knowledge. In so doing, we considered 
four quest ions: 

1. Does training sensitize students to differences among the task 
demands and information sources associated with various kinds' 
of questions? 



\ 
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2. Does training increase students 1 self-awareness of what they 

are doing' when answering questions (i.e., Can ( they better monitor 
their own performance)? 

3. Does training influence the quality of students 1 responses to 
-quest iorrs- (i.e. , Are respcms-es-~more~accurate)-? — — — 

4. Does question answering ability become more sophisticated when 
students (a) ai'e'aware of task demands of questions, and (b) can 
accurately identify the strategies they have used whi le 
answering a question? 

We predicted that as a result of training, (a) students would learn to 
invoke question answering strategies appropriately, (b) students 1 awareness 
of their own qTiest ion answering behavior would be heightened, (c) the 
quality of their responses (.■•©•-» completeness and accuracy) would improve, 
and (d) the quality of responses would improve particularly under those 
circumstances in which students were able to achieve heightened awareness 
of their own behaviors. In addition, we expected that these predicted out- 
comes would vary somewhat as a function or grade or ability levels, the 
difficulty of the material, and the task demanded by the question. 

Method 

The study consisted of two experiments. The first, using adult skilled 
readers, attempted to provide baseline data concerning the level of meta- 
cognitive awareness. exhi bi ted by expert readers when performing tasks 
related to question answering. Subsumed under th.is goal was an attempt to 
determine the minimum level of instruction required for conscious 
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metacognitive awareness of question answering strategy use. Because the 
focus of interest in this study is on the effect of intervention with 
elementfary and junior hi^gh school students, and due to the consistency of 
the adult data, the first study will be described only in a brief summary, 
fo llowed by a detai led description of the study of primary interest. For a 
rrpre complete report of the first experiment, the interested reader is 
directed to the original source (Raphael, 1981). 

The subjects in Experiment I were kk skilled adult readers from an 
introductory educational psychology course at a large midwestern university. 

Subjects participated in a ten-minute instructional period which con- . 
sisted of presenting def i ni t i ons. of three question-answer relationships 
(Pearson S Johnson, 1378) and the implied strategies for answering questions 
frxn each category. This was followed by a brief practice passage. After 
minor discussion, subjects responded to two ^00- to 600-word passages and 
(8 comprehension questions for each passage. These materials are described 
in more detail in the description of Experiment II, 

■Scores were created based upon their ability to identify the category 
from which a question was created, their use of an appropriate question 
answering strategy, and their response quality. Results strongly suggest 
that skilled adult readers can (a) easily recognize, with a minimal degree 
of instruction, the question types and their imp Lied quest ion answering 
strategies, (b) use the most appropriate strategy in responcT^g^to^ the 
quest icns, "and (c) exhibit consistency In the kinds of responses they pro- 
vide across four expository passages. In short, Experiment I served its 
function of oroviding a baseline, corroborating the assumption that skilled 
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readers possessed *>e skills in question in their repertoire of reading and 
metacogni tive abilities. 

Experiment I I 

Subjects 

One hundred students from a suburban school system participated in the 
study. Students were divided into five groups on the basis' of grade level 
or reading ability. Thus, there were three groups of average readers at 
each of the fourth, sixth, and eighth grade levels and two additional 
groups of sixth-grade students identified as low-average and high ability. 
Each group was divided into ten trained and* ten control (oriented, cf. 
p. 10) students (see Table 1). To accommodate school schedules, fourth- 
Insert Table 1 about here. 

v ""—"*" «- — —— — — —— — — — — — — — — — — 

and eighth-grade students were assigned randomly to treatment group (train- 
ing or orientation) by class (all students had the same reading teacher); 
sixth-grade students were assigned randomly to the two levels of treatment 
individually. Three criteria were applied jointly to determine reading 
ability: teacher judgment, developmental reading group membership, and 
reading comprehension scores from the Stanford Achievement Test. As a 
double check on the randomization, training versus orientation comparisons 
were carried out with each of the five subgroups, and no significant dif- 
ferences were found. 
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Design 

A 3 x 2 x 3 experimental design was used with between w subjects factors 
of grade (kth, 6th, and 8th) or ability (high, average, and low-average) 
and treatment (training and orientation). The wi thi n-subjects factor was 
question type (TE, Tl , and SI). Two cross-age and two ability comparisons 
were conducted within this design. -Within the age and ability comparisons, 
one comparison used a passage read by all students (common passage), the 
other used a set of passages from which students read the particular 
passage written at their reading level (reading level passage). That is, 
two comparisons involved average students at the fourth-, sixth-, and 
eighth-grade levels, one of which used the data from a common passage read 
by all students (e.g., "About Dogs 11 ) and the other used data from a reading 
level appropriate passage set (e.g., 4th: "Circus Clowns, 11 6thr "The 
American Cowboy," and 8th: "The Zoo Story"). Two additional comparisons 
involved sixth grade students of high, average, and low-average ability 
levels. Again, one comparison examined data from a sixth-grade level pas- 
sage read by all students; a second comparison examined data from a reading 
level appropriate passage set. * 

I nstruct ional Procedures 

The instructional program was developed using the Pearson and Johnson 

(1978)" trichotomy of question types. This categorization scheme was 
selected because it is unique in the method by which a question is classi- 
fied. While the majority of taxonomies assume that questions can be classi- 
fied :n isolation, this trjehotomy underscores the necessity of identifying 
a question type according to its relationship to both the text to which* it 
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refers and the knowledge base of the reader. Thus,* rather than speak only 
of question types, it becomes more appropriate to refer to question types 
and their implied question-answer relationships (QARs) . The three- quest ion 
types are text explicit, text implicit, and script implicit. 

A text exp1ic$^(TE) question's defined as having both the informa- - 
tion used to create the question and that used to form an appropriate 
response located within a single sentence from the text. - A text implicit 
(Tl) .question is defined as having the information used to create the 
question and that used to provide a response located, in the text, but\re- 
quir:ng the readers to integrate information across sentences, paragraphs, 
or pages. A script implicit (SI) question is defined as one which is based 
upon information in thfe passage, but which requires readers to search their\ 
knowledge base to proyide an appropriate answer. All students in Experi- 
ment II were given simplified terms to describe the three QARs (text 
explicit was called Right There; text implicit, Think and Search; script 
impl ici t, On My Own) . 

The following brief parafhiph and three questions illustrate the three 
QAR categories: 

(1) Robbi wore a scarf to school today. It was bright red. 
TE: What did Robbi wear to school today? (a scarf) 
Tl : What color was Robbi 's scarf? (red) 
SI: Why did Robbi wecr a scarf to school? (it was cold) 
During the instructional pha ses of the study, students 'eceived an 
explanation of the three QAR classifications and practice in identifying 
them. Then they received instruction concerning the implication of each 
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question type in terms of task demands, the amount or detail of the explana- 
tion varying with treatment ! g roup membership, 0 

All training was conducted with groups of students ranging in number 
from 10 to 25. While "levels of training" is used^to refer to both treat- 
ment groups, a note of explanation is necessary. To reflect the fact that 
students in the control group received an introduction into the terminology 
and task used'in the study, the label n orientat ion" group has been adopted. 
However^ it would be misleading to consider their instruction a proposed 
training level. Rather, since the orientation was identical to the minimal 
level of introduction necessary for a skilled reader (as in Experiment I) 
to perfoYm the experimental tasks, this group should be considered a control 
group. . 

Students in the orientation group received a description of the three 
QARs and one„brief practice passage with a sample quest ion Represent ing each 
of the three categories. Again, this reflects the minimum information 
needed by skilled adult readers to successfully complete the experimental 
task. 

Those students in the training group received a x four-day instructional 
program prior to doing the experimental task. On the first day, us>ing 
identical materials wi thf students at all ability and grade levels, question 
answering in general was discussed and the three question-answer 'relation- 
ships were described to provide a concepcual framework. The students then 
read passages of 30~50 words and^ responded to at least one quezt ; on from 
each*category per passage. Immediate feedback was given on both the 
selection of the categoH%^ nd the quality of the response. On the second 
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day, students were given guided practice with five grade appropriate pas- 
sages (4th, 6th, or 8th grade reading level) from 75-150 words long and wi.th 
corresponding questions from each of the three categories. On the third 
day, all students responded to a longer passage divided into four 150-word 
sections. Each section was followed by two questions from each QAR 
category. Students read and responded to questions in the first section as 
a group. The remaining sections and questions formed a criterion test to 
determine whether or not individual students had learned the three QAR 
categories. The fourth day consisted o\ individual practice for students 
who had scored below 75% on the criterion test. Only four students from the 
-f^£tJl_g.rade did not reach criterion by day three; all students reached it- 
by day four. ^ 

» 

Exper imenta 1 Task 

Students completed the experimental task after they had been exposed 
to one of the two treatment levels. The orientation group received their 
instruction immediately prior to participation in the experimental task." 
For those in the training group, the experimental task was completed on day 
five. The £ask consisted of two sub-tasks after reading a passage., of 400-^ 
600 words in ]engtb. The first task involved locating the correct response 
to a comprehension question; the second required the simultaneous classi- 

ficatioq of the question answering strategy in terms of the QAR invoWed in 

> 

locating the response information. in other words, students were asked to 

both consider how they located information to be used in. the response, as 

ft. 

well as what that information should be. The following example illustrates 
the integrated nature of the task: 
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(2) What, .is the dog often called? 

TE* * (Man's 3esV Friend) 

Tl 

SI 

-For the schooUchi ldren, the mnemonics RIGHT THERE, THINK 
£ SEARCH, and ON MY OWN replaced TE, Tl, and SI respectively. 

Scoring Scheme ^ , , 

For all responses collected in Experiments I and II, a categorization 
scheme (Raphael, et a I., I98O) developed for scoring responses to compre- 
hension questions was used. Basically, the scheme directs the person 
classifying responses to make a set of binary (yes-no) decisions to a 
sequence of questions concerning the answer location and appropriateness, 
ending up. at different terminal" nodes in a decision tree. The terminal 
nodes represent the classification of the question response. The system i'S 
quite involved; however, the reader can find a complete explanation in 
Raphael (1981). To aid in comprehension of this article, sample responses 
and their classifications wM 1 be provided. 

Assume question (2) istased upon the following text: 
(3) No matter what kind of dog it is, all dogs have one thing in 
common. They dearly love the people with whom they live. No 
wonder the dog is often called "Man's Best Friend. 11 
The answer to question (2), "Man's fiest Friend," would be classified as a 

V 

"Total Deleted Constituent." This correct response category indicates 
verbatim response from the text. A response of "Mutt'j or "Rover" would be 
classified as "Plausible Script Response" recognizing it as an acceptable 
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knowledge-abased response. The response "Echo" would be classified as an 
!l lmplausible Script Response" indicating that it is an inappropriate 
knowl edge- based response since it cannot be considered a typical name or 
label for dogs in general,. 

To validate the reliability of the scoring scheme, the percentage of 
agreement in response classification between two independent judges was cal- 
culated. A 10% sample of the test materials was systematically selected for 
use in this validation such that each combination of passages and treatments 
was represented* Though percent agreement was assessed separately for each 
passage, the results were so consistent that the data will be reported v 
collapsed across the four passages. Percent agreement was 97% with no dis- 
agreements involving a change from text to script categories, and only .6% 
involving a distinction between appropriate/inappropriate categories. 

Resul ts , » 

Four dependent measures were used to assess performance of the school- 
age students. The first, hi ts , identified the number of correct QAR 
identifications or question-answering strategy selections. The second, 
ma tches , indicated the degree of agreement between strategy ident if icatjon 
and strategy use. The third, response quality , examined the quality of the 
answer provided by the reader. The fourth, hit-match-response quality , 
indicated the students 1 ability to both recOgni ze i the task demands of the 
question and use an appropriate strategy for generating a correct response. 

Analyses will be reported by dependent measure. Within each dependent 
measure will be four sets of results: (a) comparison across average 
students in grades four, six, and eight on a common passage (hereafter, 
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Grade Comparison— Common Passage), (b) comparison across these ^ame average 
students on a passage at their level of reading abi 1 fty^heceaftfer , Grade 
Comparison— Reading Level Passage), (c) comparison across 1 ow-avenageT^ 
average, and high ability sixth-grade students on a common passage (here- 
after, Ability Compar i son--* Common Passage), and (d ) comparison of these 
sixth grade students on a passage at their level of reading (hereafter, 
Ability Comparison— Reading Level Passage). 

Analyses of variance were conducted for each measure using the factors 
of ability or grade, treatment level, and question type. Throughout this 
experiment, post hoc analyses were computed using Newman Keuls procedures. 

Means will be reported in proportions since arcsine transformations 9 
% ± 
wefe performed on all proportions but did not affect the results. An alpha 

level of .05 was used throughout the study. * However, for the convenience 

of the reader, when the £ value was significant at- the .01 level , i t wi 1 1 

be reported as such. Again, the reader is referred to Raphael (1980) for 

a complete report of the data. Due to constraints of space, the results of 

Experiment I will not be reported, and only minimum attention will be given 

to interactions which are not reacfi ly explainable or apparently due to 

"materials" effects. " 

Classifications by Question Types (Hits) 

The ANOVA for the Grade Compar i son- -^Common Passage revealed significant 
differences for grade, £(2,5*0 = 3-32, £< .05; treatment, £(1,5*0 = 31.72, 
£< .01; and question type, £(2,108)- = 9-01, £ < .01. The post hoc pro- 
cedure revealed no commonly accepted significant differences among the 
three grade levels; however, differences between the fourth and sixth, 
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£ < .0^5; and the fourth and eighth, £< .055, were very close to reaching 
standard levels of significance. Post hoc analysis of question type re- 
vealed no significant difference between text explicit and script implicit 
question ratings, but performance on both of these questions exceeded that 
of text implicit questions. Performance of. students in the training con- 
dJtjon exceeded' that of students in t|e orientation condition" (see Tabfe*^) 





Insert Table 2 about here. 



The significant treatment X question, type interaction, F_(2,108) = 3-76, 
p< .05 (see Frgure 1), revea 1 ed tha t s tud^n^ts^TrT"^^ raining groups 
exhibited" relatively similar performance across quest ion^types^ whereas 
students in the orientation condition, while lower on every question type, 
were particularly low on text implicit qiksftions." Hence:, it is likely that, 
the main effect for question type was prtmar i 1 y due to the poor performance 
of- the'orienta£ion group on text implicit questions. > There were no other 
s igni f icant i nteractions. 



I nsert°Figure 1 about here. 



The Anova for the Grade Comparison — Reading Level Passage revealed 
similar patterns of results. Main effects were found for treatment, 
1(1,5*0 = 7-97, £<-.01, and question type, F(2,103) = 3/28", p- < .05. Post 
hoc analysis revealed the same pattern of . signi f icant differences found in 
the common passage analysis. In addition, a similar treatment X question 
type interaction was significant,* F(2, 108) = 4.95, P < .01. Given that 
this analysis was based ori"data from passages at the students 1 appropriate 
reading 1 evel , Jack of a mai n ef feet for grade was. not surprising; this 

• • \ ' MS. i 
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"suggests that the treatment wa*s operating in a manner that was independent 
of the students 1 age. 

In the Ability Compari son- -Common Passage, trie AN OVA revealed signifi- 
cant main effects for ability, £(2,53) = 10.80, p_ < .01, and treatment, 
£(!,53) = 12.59, £< .OK Post hoc analyses revealed signi ficant differ- 
ences among the three ability levels, with high abilrty students performing 
at a higher level than did average who performed at a higher level than did 
low-average ability students. Subjects in t training group again scored 
at a. higher level than those in the orientation group (see Table 2). These 
effects were complicated by a significant ability X treatment X question 
type interaction, £(*t,106) = 3.73, £< .01, indicating that training dif- 
ferentially affected performance across question types (see Figure 2). 

Insert Figure 2 about here. 

Tl)e AN0VA for the Ability Comparison--Reading Level Passage reveals a 
main effect pattern similar to that found with the common gassa^e compari- 
son, with significant differences for ability, £(2,53) = 10.^1, £< .01, 
treatment, £(2, 53) = 6.07, £ < .05, and question type, £(2, 106) = ^.93, X 
£< .01. The post hoc Newman Keuls procedure revealed significant differ- 
ences between high and low-average ability students, and between high and 
average students, with no significant differences between the average and 
low-average students. This differs from the analysis in the grade-fjevel 
comparison where the grade effect was no longer significant when students 
read passages of appropriate readability for their. grade level. Instead, 
the effect of ability remained constant across the different passages read. 
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For question type, significant differences were found between text explicit 
and script implicit questions (see Table 2), the significant ability X 
question type interaction, £(4,106) - 2* 73, £< .05 appeared to be caused 
by the performance of low-average students on script implicit questions 
(see Figure 3l- Generally, students scored higher on text explicit than 

Insert Figure 3 about here. 

text implicit or script implicit questions when rating them by type. Low- 
average students, however, performed as well on the script implicit as on 
the text explicit questions, most likely explained in terms of a spurious 
"materials' 1 effect. It appears that the script implicit questions for the 
passage appropriate to their reading level were easier than for the common 

9 

passage. I 

Across the four ANOVAs, results suggest that children tended to rate 
text explicit questions with the most consistent degree of accuracy; per- 
formance on script implicit questions was^enera 1 ly near the level of text 
explicit questions. V/hile overall, performance was lowest When, rating text 
•implicit questions, students in the training group tended to rate such 
questions more accurately than did those i n" the, or i entat ion group, often 
approachi ng* pe v r formance levels ort text explicit and script implicit 
questions (see Table 2). 

Hatches: Rating^, and Responses " 

Recall that matches occur when students give responses consistent with 

c 

their classification of the type of response they thought they were giving. 
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In the Grade Compari son— Common Passage, main effects were revealed by 
the ANOVA for treatment, F(1,54) = 11.85, £< .01, and question type, 
£(2, 108) = 7-54, £< .01. Students in the training groups had a higher 
proportion of matches than those in the orientation group. Post hoc 
analyses revealed significant differences In levels of -performance between 
text explicit and script impl icit- questions, and between text explicit and 
text implicit questions (see Table 3). However, there was no significant 
difference in levels of performance between text implicit and script 
Implicit questions, though the differences were close to traditional levels 
of significance (£ < .10). There were no significant interactions. 

Insert Table 3 about here. 

The ANOVA in the Grade Comparison — Reading Level Passage revealed a 

similar pattern, though only the effect for question type was significant, 

£(2, 108) « 9.08, £< .01. The effect of treatment level, though not sig-° 

nificant (£ = .09) revealed a trend in the same direction as in the common 

passage comparison, with the training group performing at a higher level 

than the orientation group. Post hoc analyses revealed significant per- . 

♦» * 
formance differences between text explicit and script implicit and between 

text' implicit and script implicit questions, but no significant differences 
between the two text-based quest ion' types (see Table 3). No other signifi- 
cant main effects were found, nor were there any significant interactions. 

The ANOVA, for the Ability Comparison — Common Passage revealed sig- 
nificant main effects for ability, £(2,53) = 8.25, p_ < .01; treatment, 
£(1,53) = 13-91, £ < .01; and question type, £(2, 106) = 19.57, £ < .01. 

21". 



ERIC 



V 



Question-Answer ing Behav ior 
20 

No significant interactions occurred. Post hoc computations revealed sig- 
nificant differences only between the low-average and average ability - 
students, and the low-average and high ability students. The effects of 
treatment and question type*were in the same direction as those reported in 
the grade comparisons. The training ^roup performed at a higher level than 
the orientation group. Post hoc computations revealed significant dif- 
ferences between performance levels on script implicit and both text 
explicit and text implicit questions, but no significant differences in 
performance between the two text-based question types (see Table 3). 

In the ANOVA performed in the Ability Compar ison—Readi ng Level 
Passage, main effects were found for ability, F(2,53) = 3.75, p < .05; 
treatment, F(l,53) = 5.10, £< .05; and question type, £(2,106) = 13.45, 
£< .01. Post hoc analyses revealed similar patterns, of significant dif- 
ferences as in the common passage comparison with the same group. There was 
a significant difference in performance between high ability students and 
both average and low-average. The differences between the low-average and 
average students was near the conventional level of significance (.05 < 
£ < .10). Students in the training group performed at e higher level than 
did those in the orientation group* Significant differences occurred 
between script implicit questions and both text explicit and text implicit 
questions, with performances higher for text-based rather than script-based 
questions (see Table 3)* There were no significant interactions. 

Across the four analyses, when examining the number of matches as a 
function of grade or ability, treatment- level, and question type, it 
appeared that performance on text-based questions was consistently higher 

■Si . ' 

than performance on script-based questions. That is,/fo either text 
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explicit or text implicit questions, students were more likely to provide a 
response that was consistent with their perception of the question- 
answering strategy they thought was required by the question. For script 
implicit questions, the students were less consistent in their perception 
of the question and selection of appropriate question answering strategies. 
Though generally there were no significant differences in performance on 
the two text-based question types, one exception to this pattern emerged in 
the grade comparison on the common passage. In this instance only was the 
level of performance higher for text explicit than for text implicit 
questions. 

When assessed on the same passage, performance of students in both the 
grade end the ability comparisons indicated the facilitative effect of 
training over orientation. When responding to reading level appropriate 
materials while training tended to be more beneficial than the orientation, 
the difference was not consistently at a traditional level of significance. 
It appeared that greater instruction in question answering strategies led 
to more consistency between percept ion^a'nd implementation of appropriate . 
question answering strategies. While average students across the three 
grades did not differ significantly from one another in their performance 
levels on this task, sixth grade students of differing ability levels did 
vary in the predicted direction. 

Correct Response to the Questions 

The scores for this dependent measure v/ere based on the Raphael, 
et al. (1980) scoring scheme. By collapsing across scores .i n the fifteen 
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possible categories, accurate and inaccurate scores were determined. 
Response quality was categorized independently of the QAR cateqory ratings. 

An ANOVA on the Grade Compari son- -Common Passage revealed significant 
main effects for grade, F(l ,5*0 = 3-60, £< .05, and question type, 
£(2,108) = 12,82, p < .01. Ho significant effect for treatment was found 
at the conventional level. of significance, though the trend favored the 
training group (M = .87), £(1,5*0 = 3-36, £< .08, over the orientation 
(M = .82) group. For the grade effect, post hoc computations showed sig- 
nificant differences between fourth- (M = ,81) and eighth-grade (M = .89) 
students, but no differences between sixth-grade students (H = .83) and 
either fourth- or eighth-grade students. In addition, significar^ differ- 
ences for the question type effect were found between script implicit 
( M = .77) and both text explicit (M = .88) and text implicit (M = .89) 
questions. In short, the number of correct responses i ncreased -propor- 
tionally with age; and students tended to perform more successfully on text- 
based rather than .script-based questions. There were no significant inter- 
actions. 

On the Grade Comparison— Reading Level Passage,, the trends were not as 

consistent. An AtlOVA revealed significant effect for grade, F(l,54) = 4.31, 

* 

£ < .05, and question type, F(2,108) = 19.67, £ < .01. The question type 
results followed the same. pattern as that of the common story analysis, but 
the grade effect differed. The only significant difference was between 
fourth- (M = .83) and sixth-grade (M^ = .77) students; the difference between 
sixth and eighth (ft = .84) missed trad \ tional , stati st ica 1 significance by 
a small amount, £< .07. It seemed that given a passage of appropriate 
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reading level, the rate of successful performance did not follow a pre- 
dictable direction, perhaps indicating little more than the fact that'per- 
formance depends heavily on passage constraints* The treatment effect, 
while in the expected direction, did not even approach conventional sig- 
nificance, £(1,5*0 = 1.67, £< -21. 

These results were complicated by the two-way interaction of grade X 
question type, £(^,108) = 3.69, £< .OK it appeared that in the fourth 
and the eighth grades, more correct responses were given to tQxt explicit 
than text implicit questions, and both text-based question types had a 
higher proportion of correct responses than did the script implicit 
questions. However, in the sixth grade the patten, differed. While 
students performed at a^vJgher level on text than on script questions, they 
tended to be more successful with text implicit tfian text explicit questions 
(see Figure h) . Again, the only explanation seemed to be passage specific 
differences: it may be the case that for the sixth grade 1 eveT readabi \\ ty 
passage, text implicit questions were inherently easier' than were the text 
implicit questions for the other three passages, possibly as a function of 
degree of topiq familiarity. 

Insert Figure k about here. 

The ANOVA for the Ability Compar i son—Common Passage revealed signi- 
ficant main effects for ability, £(2,53) = 20.01, £< .01; treatment, 
£(U53) = **.09, £< .05; and question type, £(2,106) = 37.80, £< .01. 
Post hoc analyses reveale that all pair-wise comparisons for these three 
variables were significant. For the ability effect, the proportion of cor- 
rect responses achieved vyas in the predicted direction with high ability 
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students (M = .91) exceeding average (M = .77), and average exceeding low- 
average (M = ,60) students. The treatment effect again demonstrated the 
superior performance of the training group (M = .80) over the orientation 
groyf) ( M = .72). Students scored higher on the text explicit (M = .77) 
ar>d text implicit (M = .35) than on the script implicit (M = .66) questions, 
5/nd higher on the text implicit than on text explicit questions. 

The significant ability X question type interaction, £(4,106) = 2.41, 
£< .05 (see Figure 5), revealed a similar ranking of question types across 
all, ability levels, with the probable source of the interaction being the 
extremely low performance on script implicit questions by the, low-average 
students. This was further complicated by a significant three-way 
interaction between ability? treatment, and question type. V/hile the 



Insert Figure 5 abwut here. 



general pattern remained unchanged," in the training condition the njagnitude 
of the dif-fprences in perfornvMce levels of the three ability* leyels wgs > 
smaller than the magni tude x of the differences in performance levels between 

o 

the three ability levels in th£ orientation group. The source of the Inter- 
action was due to the /act that the genera 1 ; pattern of higher performance 
on text than script, and on -text implicit over text explicit held for all 
groups except the trained high ability students, for whom the training led 
to an even level of performance across all question types (see Figure 6). 



Insert Figure 6 about hen*. 
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An ANOVA for the Ability Comparison—Reading Level Passage revealed 
similar significant main effects: ability, JF (2 , 53) 55 k.2k y £< ,05; treat- 
ment, F( 1 , 53) = 6,29, £< .05; and question type, £(2,T06) = 21.61, , 
£ < .01. Post hoc analyses, revealed no significant differences betweerj the 
low-average ( M = .79) and the average students (M = ,77), However, the high 
ability students (ft- ,87) pei formed significantly higher than the other two' 
ability groups. Post hoc analyses also revealed significant differences 'in 
performance as a function of questipn type only between script implicit 
( ft = .71) and either the text explicit ( M = .86) or text implicit (M^ = ,86) 
question type. Students provided more correct responses to either type of 
text question than to those" 'scri pt-based ones. Consistent with previous 
findings, the training group (M = '.85) performed at a significantly higher 
level than did the orientation group (ft = .77). The/signif icant ability X 
question type .interaction, F(4,106) = 2.60, £< .05, revealed that while 
both high and average ability %tudents exhibited a gradually decreasing 
trend across the'TE - TI - SI trichotomy, low-average students scored 
higher on the text implicit than on text explicit questions -(see Figure 7). 

I nsert Figure 7 about here. 

— --* - ~- r -- — •« 

Across the four ANOVAs for response quality, the question type effect' 

. • ■ i 

was generally consistent. Correct responses occurred more frequently on 

text; than on script questions, with little difference between performance 

on text explicit and text implicit questions. When compared on a common 

passage, the effect of ability was in the expected direction as was the 

grade effect. However, this pattern was not replicated in the comparison 

of reading level appropriate stories. . V/h i le high ability students tended 
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to provide more correct responses than did average or. low-average, the mag- 
nitude of the difference was less than in the common story comparison. In 

the comparison of, fourth-, sixth-, °and eighth-grade students, using the 

I 

reading level passage set, fourth- and eighth-grade students provided more 

Q 

correct responses than did sixth-grade students. This is likely to be due 
to a materials effect similar to that mentioned earlier, since comparison 
on a common passage revealed response patterns in the expected directions. 

Where the treatment level' was significant, the training group per- 
formed at a higher level of success than did the orientation group, a trend 
that existed even when the effect did not reach a conventional level of 
statistical significance. For example, in the eighth grade comparison, 
common passage, the level of performance was higher for the training group, 
.05 < £ < .08. 

In summary, with the exception of those responses which involved text 
impl icit. questions in the sixth-grade passage, performances were consistent 
with respect to findings on the first two dependent measures. Text 
questions^of either type elicited higher 'level s of performance than did 
script questions, and training was more facilitative than was orientation. 
The results for the treatment effect suggest* training was particularly 
effective with high ability students on their responses to script questions 
and with low-average and average students in their responses to text-based 
questions. 

Correct Hi t-Match- Response Quality 

Recall that scores for this dependent measure were created by identi- 
fying cases in which students who received both a "hit" and a "match" on 
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a question also provided an answer from one of the accurate response cate- 
gories. The ANOVA for the Grade Comparison—Cormion Passage revealed 
significant effects for treatment, £(.1,5*0 - 32-65, £< .01; and question 
type, F(2,108) = 13-72, £< .Ok Training (M = .73) was superior to 
orientation (M^ = .52). Post hoc analyses revealed no significant differ- 
ences between text explicit ( M = .69) and script implicit questions 
(M = .69), but bpth were significantly better than' text implicit (M = .^9). 
The significant treatment X question type interaction was similar to that 
in the Reading Lev^l Passage comparison and wi 1 1 be discussed later. 

(The ANOVA for ^(ie Grade Compari son— Read i ng Level Passage revealed the 
same pattern. The effects for treatment, F(l,5'0 » 10.15, £< .01; ana for 
question type, £(2,108) = 6.78, £< .01, were significant. Training 

( M = .67)' was again superior to orientation (M = .53). Post hoc analyses 

\ 

revealed no significant differences between text explicit (M .62) and 

\ 

script implicit questions (M = \67) , but bbth differed significantly from 
text implicit (M = .52). 

. The significant treatment X question type interaction (see Figures 
8 and 9) revealed similar patterns across comparisons. In both cases, 
students in the training group scored higher than those in the orientation 
group on all three types of questions; however, the between group differ- 
ence was much larger for text implicit than for either text explicit or 
script implicit questions. Generally, the training appeared to greatly 

Insert Figures 8 and 9 about here. 
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facilitate the students 1 awareness of the need to integrate text informa- r, 
tion, thus improving their ability to cope with this type of question. 
Overall, it appeared that orientation was : most successful in improving 
performance on text explicit and script implicit questions, though even in 
these two categories, performance improves with the kind of additional 
training provided by the treatment. 

An AN OVA on the Ability. Comparison—Common Passage revealed signifi- 
cant effects for ability, £(2,53) = I3.5*,._£< .01, and treatment, £(l ,53) = 
19.16, £< .01. Post hoc analyses revealed significant differences across 
the three level,s of ability; again, the higlj ability students (M^= .70) 
putperformed the average (M = .56) who outperformed the low-average ability 
students (M = .i*2). The training group (M = .6$)- scored higher than the' 
orientation group ({1 = .47). • ; 

These results were 'complicated b<y a signi f i cant abi 1 i ty. X treatment X 
question type int4raction F(A,106) = 3-08, £ < .0?. Training, as opposed 
to orientation, facilitated high ability students 1 performances on both 
text explicit and script implicit questions. Yet the training did not 
appear, to improve performance on..taxt implicit questions to any larger 
degree than did orientation. Within the average ability students cell, 
training raised the level of these students to the level of the high 
ability students in the, or ientatioiji group on all three question types. With 
the low-average students, tra i ning/ seemed most faciiitative on text-based 

questions, parti cu 1 axJ y t ex t exp L) c "i t ones "(Tee TFjgure~To)~ ~ 

/ 

Insert Figure 10 about here. 

/ 
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The ANOVA in the Ability Compari son--Readi ng Level Passage revealed 
n effects for ability, F(2,53) =8.77, p < .01; treatment, F(l,53) = 
11.97, £. < .01; and question type, F(2,106) =: 10.07, < .01. Post hoc 
analyses revealed significant differences between the low-average (M = .53) 
and the high (]1 = .71) ability students, and between the average (tt = .56) 
and the high. Significant differences occurred between text implicit 
( M - .k3) and text explicit ( = .6*0, apd between text implicit and script 
implicit ( H = .65) question types. There was no significant difference 
between the script implicit and the text explicit question types. It 
appeared that, in general, students with passages at their reading level 
performed at a lower level on text implicit questions than they did on the 
common sixth-grade passage. This' argues for a "materials" effect exp^lana- 
tion once again. Students in the training group (M = .66) performed at a 
higher level than those in the orientation group 0^= .53). 

The significant ability X question type interaction, F_(l|,106) = 3-53, 
£< .05, indicated that while text implicit questions were the most diffi- 

T 

cult for students of a!! levels of ability, the patterns of_ performance on 
text explicit and script implicit quest^p^s, and the magnitude of the pair- 
wise differences on all question types, varied across the three levels of 

V! 

ability. High ability students performed highest on te xt ftxplir.it- 



questions, and only slightly higher on script implicit than on text 
implicit questions. Average and low-averag£ students general 1 y "fol lowed 
a SI > TE > TK pattern. However, for the average students, the magnitude 

tr 

of the pairwise differences among the three question types was quite small, 
while for the low-average it was large. For the low-average students, 
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performance differences between text explicit and script implicit questions 
were smaller proportionally than performance differences between TE-Tt and 
SMI comparisons (see Figure 11). 

.Insert Figure 11 about here. 

Across the four analyses for this dependent measure, the results 
generally favored the training over the orientation group. This was true 
% in all cases except high ability students responding to text implicit 
questions. This was"; the only situation i n \%rh i ch training seemed no more 
facilitative than orientation. 

In contrast to the first three .dependent measures, performance on 
script impl ici t A questions was 'generally higher than that on text implicit 
questions, and often equal to that on text explicit questions. . It seems to 
be the case that students can access an acc'urate answer to a script 

implicit question as easily as a text explicit one rf they first recognize 
the strategy opt.imal for responding to the question. That* is, if they 
realize that they shou<ld depart from the text to answer the question, they 
are as likely to give an appropriate response as they are when they recog- 
'nize that they should consult the text. 

Pi scuss ion 

The first question of concern was the degree to which training would 
sensitize students to differences among the task demands and information 
sources appropriate to various kinds of questions. We predicted that 
students would le v arn the system taught them and hence would display greater 
sensjjti vi fry to the relationship between questions and strategy use in 
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locating appropriate responses. In other words, the tra i ni ng shou Id have 
better enabled students to identify the type of QAR most readily invited 
by a particular question, and this is precisely what occurred, albeit with 
differing patterns in different analyses. 

Adults who received a brief orientation performed with near perfect 
accuracy. While accuracy overall was lower than for the adults, average 
fourth-^ sixth-, and eighth-grade students in the training group consis- 
tently rated the questions from each category more accurately than students 
in the orientation group, most nptably on text implicit questions. Ability 
tended to affect performance in the expected direction, with the high 
ability students performing the most successfully, followed by average and 
low-average students. However, when examined ; n terms of training levels 
and QAR categories, ability differentially enhanced students 1 > performances 
on each question type; yet, the different patterns appear td be most likely 
due tb spurious materials effects. Nonetheless the overall pattern, even 
within* interactions, was for trained students to exhibit better QAR ratings 
than oriented students., 

The second concern of the study was the degree to which training would 
sensitize students to their own question-answering behaviors. We pre- 
~uiuLed thai trained scuaents would become better monitors of their own 
performance, thus achieving a higher proportion of matches (providing 
responses which matched their QAR rating, independent of response quality). 

The elementary and junior high school students again performed at a 

generally' lower level than did the adults, with the students in the training 
group performing at a higher level than those in the orientation group. 
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This pattern was consistent across both age and ability when students were 
compared on a common story. When given passages at an appropriate reading 
level (regardless of grade level) a somewhat different pattern emerged. 
Both training and orientation groups performed at the same level on script 
implicit questions, but the training group performed at a higher level than 
did the orientation group on both text-based questions. All students in 
the training group showed greater consistency in identifying questions as, 
text explicit and text implicit and locating the appropriate text-based 
information for their response than they did when the questions were script 
based. , It was easier for students, to provide a text-based response to a 
QAR they had rated as text-based (TE or TI) than it was to provide a 
scriptal response to a QAR they had rated as scr i pt-baseS. This supports- 
findings such as Guszak's ( 1 966) and more recently, Chou-Hare and Pulliam 
(1980) that students have an easier time handling literal (i .,'e. text- 
based) questions. Nonetheless, the second prediction was also sub- 
stantiated: training consistently enhanced performance in comparison to 
an orientation; however, there were variations in this pattern across 
question types and some minor variation as a function of grade rjnd ability 
level s. • ' - 

TTre chi rcFquest ion of concern was zhe x degree to which training 

enhanced the quality of the students 1 responses. We predicted that trained 
students would give better quality answers to the questions, quite inde- 
pendent of whether they a! so achieved hits^ and matches. The pattern of 
results suggests that the evidence for the training effect is less 
impressive here than for the preceding dependent measures: In only two 
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of the four analyses did a training effect emerge, although it approached 
conventional levels of significance on the other analyses* 

The training effect was complicated by several interactions* Training 

.had differential effects for each ability level. For I igh ability students, 
the effect of training ov$r orientation was largest when they -responded to 
script impl icit" questions* The average and low-average students appeared to 
benefit most by training iri responding to both text-based questions. The 
data suggest that a still longer trafning period may be necessary to improve 
performance for low-average and, average student on script implicit 
questions. There may be a threshold effect in that the high ability 
students were performing quite successfully on the .tex^-based questions 
and the- training may have merely increased their sensitivity to the value 
of their knowledge base. The students of lower ability improved on text- 
based questions, but .perhaps need further training in the use of the 
information in their knowledge base. > 

In addition, when given a passage common across all grade or ability " 
levels, the expected ability and developmental trends occur. However, when 
given a passage at the appropriate reading or grade level, developmental 
differences tended to either not reach a significant level or to decrease 

"In che magni tu de~of~l:he diTf erences. Though tKe~~g rade level dirrerences — 
were not stable across passage of different readability levels, ability" 
differences remained constant and in the expected directions regardless of 
passage level read. 

In summary, the findings suggest that ove/all, text-based QARs are 
easier for students to cope with than script-based QARs and training 
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appears to facilitate performances differentially across ability levels and 
question types. 

\ The foirrth question evaluated the degree to which correct application 
of strategy selection procedures :hat were part of the instruction would 
enhance the ability to respond to qjestions. We predicted that the response 
quality differences between trained and oriented students would be accen- 
tuated in precisely that subset of situations in which student judgments • 
sbout the strategies they had used corresponded to the system they learned 
(were hits) and were consistent with the kinds of responses they actually 
gave (were matches). 

Results again, in general, favored training over orientation with 
ability level affecting performance in the expected direction. -It was 
interesting to note that while ability had an effect, thetje appeared to be 
no developmental effect across the three grade levels; yet all grade school 
students performed with less accuracy (had a higher proportion of incorrect 
responses) than did the adults. 

In all analyses, performance was least successful on text implicit ■ 

questions, and it is with these questions that training appeared to have 

t' 

its greatest effect over orientation. Across all grades , "those students 
-vrfio-r-ecei v e d —tr-effrti^-per formed-at-a— hfgher— heve \ overall than did those i n 

the orientation group; training resulted in performance levels on text 

implicit questions equal to those of both text explicit and script implicit 
-questions. —Thi s- occurred -for— passages-common- to t-he-grade-or— abH-i ty- — — — 

levels as well as those of the appropriate rea"ding level. 
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When examining differences as a function of ability, the same general 
patterns emerge with a few interesting exceptions. For example, training 
did not appear to enhance the performance of high ability students on /text 
implicit questions to a greater degree than did orientation. Yet, trailing 
did significantly improve their performance on the text explicit and script 
implicit ones. Perhaps further training on Tl questions could involve 
explicit directions for methods of text integration. 

The pattern of performance of the low-average students is different on 
the fourth measure than on previous ones* Their level of accuracy for 
script implicit questions exceeds or differs only slightly from both text 
explicit and text implicit questions, suggesting that if they recognize a 
question as requiring a script-based response and go to their knowledge 
base to locate it, they perform at a higher level of accuracy than when 
responding to a tc.*t-based question. This appears to support the training 
of these students for in identifying the QAR and selecting the matching 
question-answering strategy, they completely alter and surpass previous 
performance levels. 

Thus, it appears that the fourth prediction was confirmed. Perfor- 
mance is most enhanced when students both recognize the QAR most readily in- 
vited by t he question amJ use an dpp i up t idle ques tta n-answer i ng strategy,; 

this generalization, like the others, is qualified by context-specific 
effects imposed by question type, passage type, and ability. 

The literature concerning questions suggests that (a) there are dif- 
ferent types of questions, each Requiring different cognitive operations, 
(b) questions are an i nherent -part of the educational environment, 
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(c^ questions are an effective tool for enhancing learning, and (d) we can 
learn how information is processed by studying how questions are answered. 
The results of this study lend support in varying degrees to each of these 
points. 

The work with taxonomies has been largely non-empi r ica K A few studies 
(e.g., Chou-Hare & PulUam, 1980; Washington, 1979; Wixon, I98O) provide 
evidence that the Pearson and Johnson (1978) taxonomy can be operat iona 1 ized 
for both question categorization and question creation. The present study 
extends this work, providing data to support the idea that questions can be 
created within each category and that overall, adult skilled readers and 
students can be taught consistency in identifying QARs and selecting appro- 
priate strategies for responding to the question. Consistent with Wixson f s 
(1930) and Frase's ( 1 968) findings, text implicit and script implicit 
qyestions were more difficult than text explicit questions. Consistent 
with Hansen's ( 1 98 1 ) results, children generally performed better on text 
t-han script questions. 

The metacognitive research, particularly work in metamemory, suggests 
that students can b* made aware of strategies used during a variety of cog- 
nitive tasks, and that important in learning any skill is noi cr.ly the 
knowledge or, but also the control over, a given process. Research (e.g. , 
Ryan,.193l; Brown & Campione, 1977; Golinkoff, 1 975^76; Olshavsky, 1976-77) 
has indicated that successful use of strategies is highly dependent upon 
the ability and age of the iearner. The resul ts-of~this" study re consis- 
tent with these findings, extending this work into the area of question 
answering strategies. 
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Of the sixteen analyses conducted In this study, training groups per- 
formed at a higher level than did orientation groups throughout, usually at 
a conventional level of signif icance. This provides reasonably strong 
support for the position that the use of question answering strategies is 
a trainable metacogni tive skill. Generally, consistent with Day (1980) and 
Gordon (1980), it appeared that training in metacogni t i ve skills is gen- 
erally faci 1 itative, and that more explicit training is superior to less 
expl Kit instruction. 

Performance differed in the expected directions as a function of age 
and ability when students were compared on a cormion passage (i.e., an: 
identical passage and question set for comparisons of grade or ability 
levels). 'An important difference in patterns emerged, however, when 
students were given a passage of appropriate readability. The differences 
between levels of ability when compared on the level-appropriate passages 
were smaller than had existed on the common passage, and at times did not 
reach the conventional level of significance, The contrast was similar 
when comparing across grade levels. While differences in performarce 
existed across grade levels on the common passage, this rarely occurred 
when passages of appropriate reading level were used. This suggests that 
older or more mature readers using the same strategies, and texts as the 
younger or less skilled readers' are not necessarily required to do what 
appears to be the same task. The more ecologically valid task may be to 
use materials at appropr iate reading and grade levels. 

In fact, one finding of this study worth noting stems from the 
selection of the .two passages to be used throughout the second experiment. 
Recall that there were two basic comparisons among levels of students. The 
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first involved a developmental comparison using only average students -jn 
fourth, six\<, and eighth grades. The second, an ability comparison, had 
as subjects only ?ixth grade students of low-average, average, and high 
reading abilities. in designing the study, major consideration was given 
to the material upon which the students would be compared. Some (e.g., 
Taylor, 1980) have argued that content must be held constant, but that the 
actual readability of the passage should be^at the reading level of the - 
student. Others (e.g., Tierney, Bridge, S Cera , J978-79) have cautioned 
that it may be methodologically unsound to draw comparisons across different 
passages, that this confounds any effect wPth materials. Both arguments 
seemed to have merit; th.us one common and one level appropriate passage were 
selected for* each comparison, fo maximize the comparability of the dif-, 
ferent passages to used, extensive norming and pilot studies were 
conducted; in addition, the first experiment indicated that the four pas- 
sages were comparable for ski 1 led readers. 

When performances were compared within abilit/ and age levels and on ' 
treatment and question type effects across analyses of the two passages, 
predictable patterns emerged. The expected developmental trend occurs 
when aT students respond to a common passage of a fourth grade readability 

level: 3 > 6 > h. However, when students reacted to a*reading fevel appro- 

> 

prlate passage, the patterns were not as consistent. For example, there 
was not developmental trend for frits or matches. 

However, this is not necessarily the case when ability comparisons are 
drawn. ^The effect of ability was more stable across passages, reaching 
significance on all four of the common and all four of the reading level 

4U ' • 



Quest ion-Answe ring Behavior 
39 

passage ANOVAs. Nonetheless, while the effect of ability remained whether 
one used a common or reading level appropriate passage, the discrepancy 
between the pairwise comparisons decreased when the students responded to 
passages of their reading level.- 

* it appears that there is no simple solution to the problem of 
materials to be used in developmental or cross-ability studies, Reading 
level passages confound content with grad£ or ability level . Common pas- 
cages pl^ce students of differing ability in different difficulty situations. 
Perhaps all that can be suggested at this point is a caution against inter- 
preting significant differences in performance as being due to either 
ability, age, or task content until the effect is replicated with both 
types of passages. 

Finally, this study lends support to the criteria for successful 
training studies suggested by Brown, Campion^, *and Day ( 1 98 1 ) and listed 
earlier in this paper. The skill was i nstruct ional 1 y relevant as suggested 
by the question literature (e.g., Anderson S Biddle, 1975; Guszak, 1966; 
Lehnert, 1979; Rothkopf, I966; Wixson, I98O). Training proceeded ' from the 
simple to the complex (recall training procedures, cf. pp. 10-11). An 
analysis of the training and transfer tasks. (Raphael , et al., 1980) pro- 
vided evidence of where breakdowns could occur. Di rect . i nstruct ion about 
strategy use along with immediate feedback during, the class discussions 
and after individual work improved performance. A variety of passages were 
used to facilitate transfer of the strategy use to new situations, and 
finally, self-checking procedures were an inherent part of operational izi ng 
the text explicit—text imp 1 ici t--scr ipt implicit distinction. Students 
were taught to monitor their success in selecting a strategy by using the 
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Right There, Think and Search, and On My Own trichotomy (cf. pp. 10-11). 
That the guidelines suggested by Brown, et al. were implemented and vali- 
dated indicates that their instructional model has promise for future 
instructional design. * 

There were several factors which suggest limitations in interpreting 
the present data. For example, the sample of elementary and junior high 
school students was from a school system whose students typically performed 
above the national norms on standardized tests. Thus, each group of 
students was performing approximately one grade equivalent higher than 
"normal." A second limitation results from the absence of a "no- treatment" 
control group, although it would seem reasonable to expect that comparisons 
between treatment and no-treatment control should exceed those found 
between training and orientation groups, Jhird, all groups w^re taught by 
the experimenter rather than the classroom teachers for methodological 
reasons; hence there is no evidence that° the strategy instruction could be 
incorporated easily into the cjassroom academic curriculum. 1 Finally, even 
with the extensive norming procedures, material effects were suggested by 
the patterns of performance on text implicit questions in t(ie sixth grade 
passages, and by the difference in performahce level of the fourth grade 
students on the two fourth-grade passages. 

A similar study could be conducted using unfamiliar materials to 
further examine the role of familiarity, particularly with reference to 
performance on script questions, and t|?e degree of students 1 reliance on 
the text. Additional factors to be considered could include the effect of 
text access (being able to look back or not), increased levels of training 
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(as well as the use of a no- treatment control group), and differential 
levels of feedback. Another logical step would be to train teachers to use. 
the techniques as part of their ongoing curricula in reading, social 
studies, or science classes. It is one thing to demonstrate the power of a 
factor; it is quite another to demonstrate that it can be easily incorpo- 
rated into curricula that already suffer from an excess of demand- 
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Table 1 ' 

Subject Placement, Experiment II 



Abil ity 



*ith 



Low- Ave rage 



Grade 



6th 



10 Training 
10 Orientation 



8th 



Total s 



20 



Average 



10 Training 



10 Training 



10 Training 



10 Orientation 10 Orientation 10 Orientati 



60 



on 



High 



10 Training 
10 Orientation 



20 



Totals 



20 



60 



20 
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Table 2 

Table of Means of Proportion of Hits: Experiment II 



Factor/Level of Factor 



AbM ity 

• High 
Average 
Low-Average 

Grade. 

4th 
6th 
8th 

Trea tment 

Training 
Orientation 

Question Type 

Text Expl ; ci t 
Text Implicit 
Script Impl icit 



Grade Comparison 

Read i ng 
Common Level 
Passage Passage 



.60* 

.69 

.69 

.75** 
.57 



• 73** 

.57 

.69 



.61 
.60 
.68 



.69** 
.58 



.65* 

• 57 

• 67 



Abil ity Comparison 

Reading 
Common Level 
Passage Passage 



.71** 
.60 
• 50 



.67** 
■ 50 



.63 
,60 
■ 59 



.60 

• 57- 



.68* 
.60 



.69** 
• 56 
.66 



*Significant difference, £< .05 
**Signif icant difference, p < .01 
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Table 3 

Table of Means of Matches: Experiment II 





Grade Comparison 


Abi 1 i ty Compar 1 son 


Factor/Level of Factor 


Common 
Passage 


Read! ng 

Level 
Passage 


Common 
Passage 


Read i ng 

Lev^l 
Passage 


Abi 1 i ty 










High 






. B9** 


.89** 


Average 






84 


.84 


Low- Ave rage 






71 
• / I 


78 


Grade 






• 




4th 


.89 


.88 




■>«. 


6th 


.88 


.84 






oth 


.90 


.83 






Treatment 










Training 


.93** 


.87 


.88** 


.86* 


Or ientat ion 


.84 


.82 


.74 


.80 


Question Type 










Text Expl ici t 


.93** 


.87** 


.85** 


.88** 


Text Impl ici t 


.88 


.88 . 


.87 


.86 


Script Impl ici t 




• 79 


.71 


.75 


"Significant column dif 


r erence, p < 


.05 







Quest ion-Answering Behavior 

Figures 

Figure 1. Graph of the treatment X question type interaction for the 
proportion of Hits for the Grade Comparison — Common Passage, 

Figure 2. Graph of the ability X treatment X question type interaction 
for the proportion of Hits for the Ability Compar i son—Common Passage. 

Figure 3- Graph of the ability X question type interaction for the 
proportion of Hits for the Ability Comparison — Reading Level Passage Set. 

Figure *f. Graph of the grade X question type interaction for the pro- 
portion of correct responses for the Grade Comparison — Reading Level 
Passage Set. 

v ^Figure 5. Graph of the ability X question type interaction for the 
proportion of correct responses for the Ability Comparison—Common Passage. 

Figure 6. Graph of the ability X treatment X question type inter- 
action for the proportion of correct responses for the Ability Comparison — 
Common Passage. 

, Figure 7. Graph of the ability X question type interaction for the 
proportion of correct responses in the Ability Comparison — Reading Level 
Passage Set. 

Figure 8. Graph of the treatment X question type interaction for the 
proportion of correct Hit-Match-Response Quality for the Grade Comparison — 
Common Passage." 

Figure 9. Graph of the treatment X question type interaction for the 
proportion of correct Hit-Match-Response Quality for the Grade Comparison — 
Reading Level Passage Set. 
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Figure 10. Graph of the ability X treatment X question type inter- 
action for the proportion of correct Hit-Match-Response Quality for the 
Ability Comparison — Common Passage. 

Figure 11. Graph of the ability X question type interaction for the 
proportion of correct Hit-Match-Response Quality for the Grade Comparison- 
Reading Level Passage Set. 
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